Towards Informed Exploration for Deep Reinforcement Learning

In this thesis, we discuss various techniques for improving exploration for deep reinforcement learning. We begin with a brief review of reinforcement learning (RL) and the fundamental v.s. exploitation trade-off. Then we review how deep RL has improved upon classical and summarize six categories of the latest exploration methods for deep RL, in the order increasing usage of prior information. We then explore representative works in three categories discuss their strengths and weaknesses. The first category, represented by Soft Q-learning, uses regularization to encourage exploration. The second category, represented by count-based via hashing, maps states to hash codes for counting and assigns higher exploration to less-encountered states. The third category utilizes hierarchy and is represented by modular architecture for RL agents to play StarCraft II. Finally, we conclude that exploration by prior knowledge is a promising research direction and suggest topics of potentially impact

Scar/WAVE complex suppresses cell invasion and cancer cell transformation

The mechanisms by which cancer cells hijack the actin cytoskeleton to invade and disseminate to distant sites of metastasis remains one of the great frontiers in cancer research. Many actin-regulating proteins have been identified to be important in cancer cell invasion and metastasis. However the role of a major actin assembly promoting complex, Scar/WAVE regulatory complex (WRC) in cancer cell invasion is poorly understood. WRC has a well-known motility-promoting role in 2D planar cell migration, but a recent study on human epithelial cancers suggests WRC may be anti-invasive in vivo. To investigate the controversy, human epithelial cancer cells with reduced WRC expression were tested in multiple 3D cell motility assays. Interestingly, WRC demonstrates a robust anti-invasive role in these exciting experiments. To understand how loss of WRC promotes invasion, the molecular mechanism is investigated. N-WASP is the other major actin assembly promoting protein. Unlike WRC, N-WASP is interestingly not required for 2D planar cell migration, but is important for motility in 3D. The interplay of the two major actin assembly promoting proteins has not been explored in 3D cell motility. I report here that loss of WRC promotes hyper-activation of focal adhesion kinase that leads to N-WASP accumulation and activation at the invasive front. This chain of events results in enhanced invasion providing a molecular mechanism of WRC’s anti-invasive function.  In addition to this FAK-N-WASP core mechanism, I also identified a novel pro- invasive role of HSPC300 independently of WRC. Loss of WRC possibly releases free HSPC300 that could subsequently interact with and regulate N-WASP activation during invasion providing a potential direct molecular link between the two proteins. Furthermore, WRC also supresses focal adhesion kinase mediated cell transformation and tumour formation in vivo. In this thesis I therefore demonstrate novel anti-invasion and anti-tumourigenesis functions of WRC. I also show how a novel WRC binding protein, NHS, could negatively regulate WRC function

Gaussian Process Regression for Prediction of Sulfate Content in Lakes of China

In recent years, environmental pollution has become more and more serious, especially water pollution. In this study, the method of Gaussian process regression was used to build a prediction model for the sulphate content of lakes using several water quality variables as inputs. The sulphate content and other variable water quality data from 100 stations operated at lakes along the middle and lower reaches of the Yangtze River were used for developing the four models. The selected water quality data, consisting of water temperature, transparency, pH, dissolved oxygen conductivity, chlorophyll, total phosphorus, total nitrogen and ammonia nitrogen, were used as inputs for several different Gaussian process regression models. The experimental results showed that the Gaussian process regression model using an exponential kernel had the smallest prediction error. Its mean absolute error (MAE) of 5.0464 and root mean squared error (RMSE) of 7.269 were smaller than those of the other three Gaussian process regression models. By contrast, in the experiment, the model used in this study had a smaller error than linear regression, decision tree, support vector regression, Boosting trees, Bagging trees and other models, making it more suitable for prediction of the sulphate content in lakes. The method proposed in this paper can effectively predict the sulphate content in water, providing a new kind of auxiliary method for water detection

Increasing but Variable Trend of Surface Ozone in the Yangtze River Delta Region of China

Surface ozone (O-3) increased by similar to 20% in the Yangtze River Delta (YRD) region of China during 2014-2020, but the aggravating trend is highly variable on interannual time and city-level space scales. Here, we employed multiple air quality observations and numerical simulation to describe the increasing but variable trend of O-3 and to reveal the main driving factors behind it. In 2014-2017, the governmental air pollution control action plan was mostly against PM2.5 (mainly to control the emissions of SO2, NOx, and primary PM2.5) and effectively reduced the PM2.5 concentration by 18%-45%. However, O-3 pollution worsened in the same period with an increasing rate of 4.9 mu g m(-3) yr(-1), especially in the Anhui province, where the growth rate even reached 14.7 mu g m(-3) yr(-1). After 2018, owing to the coordinated prevention and control of both PM2.5 and O-3, volatile organic compound (VOC) emissions in the YRD region has also been controlled with a great concern, and the O-3 aggravating trend in the same period has been obviously alleviated (1.1 mu g m(-3) yr(-1)). We further combined the precursor concentration and the corresponding O-3 formation regime to explain the observed trend of O-3 in 2014-2020. The leading O-3 formation regime in 2014-2017 is diagnosed as VOC-limited (21%) or mix-limited (58%), with the help of a simulated indicator HCHO/NOy. Under such condition, the decreasing NO2 (2.8% yr(-1)) and increasing VOCs (3.6% yr(-1)) in 2014-2017 led to a rapid increment of O-3. With the continuous reduction in NOx emission and further in ambient NOx/VOCs, the O-3 production regime along the Yangtze River has been shifting from VOC-limited to mix-limited, and after 2018, the mix-limited regime has become the dominant O-3 formation regime for 55% of the YRD cities. Consequently, the decreases of both NOx (3.3% yr(-1)) and VOCs (7.7% yr(-1)) in 2018-2020 obviously slowed down the aggravating trend of O-3. Our study argues that with the implementation of coordinated regional reduction of NOx and VOCs, an effective O-3 control is emerging in the YRD region.Peer reviewe

A Phase-Coded Time-Domain Interleaved OTFS Waveform with Improved Ambiguity Function

Integrated sensing and communication (ISAC) is a significant application scenario in future wireless communication networks, and sensing capability of a waveform is always evaluated by the ambiguity function. To enhance the sensing performance of the orthogonal time frequency space (OTFS) waveform, we propose a novel time-domain interleaved cyclic-shifted P4-coded OTFS (TICP4-OTFS) with improved ambiguity function. TICP4-OTFS can achieve superior autocorrelation features in both the time and frequency domains by exploiting the multicarrier-like form of OTFS after interleaved and the favorable autocorrelation attributes of the P4 code. Furthermore, we present the vectorized formulation of TICP4-OTFS modulation as well as its signal structure in each domain. Numerical simulations show that our proposed TICP4-OTFS waveform outperforms OTFS with a narrower mainlobe as well as lower and more distant sidelobes in terms of delay and Doppler-dimensional ambiguity functions, and an instance of range estimation using pulse compression is illustrated to exhibit the proposed waveform\u2019s greater resolution. Besides, TICP4-OTFS achieves better performance of bit error rate for communication in low signal-to-noise ratio (SNR) scenarios.Comment: This paper has been accepted by 2023 IEEE Globecom Workshops (GC Wkshps): Workshop on Integrated Sensing and Communications for Internet of Thing